{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://img-blog.csdn.net/20180414103300419)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "解码是seq2seq模型的常见问题，常用方法有贪心搜索`（Greedy Search）`集束搜索`（Beam Search）`。\n",
    "\n",
    "Decoder根据Encoder的中间语义编码向量c和`<s>`标签得到第一个输出的概率分布`[0.1,0.1,0.3,0.4,0.1]`，选择概率最大的`0.4`，即`moi`。\n",
    "\n",
    "根据隐向量h1h1和moi得到第二个输出的概率分布`[0.1,0.1,0.1,0.1,0.6][0.1,0.1,0.1,0.1,0.6]`，选择概率最大的`0.6`，即`suis`。\n",
    "\n",
    "以此类推，直到遇到`<\\s>`标签，得到最终的序列`moi suis étudiant`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 集束搜索"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "上面的贪心搜索只选择了概率最大的一个，而集束搜索则选择了概率最大的前k个。这个k值也叫做集束宽度（Beam Width）。\n",
    "\n",
    "还是以上面的例子作为说明，k值等于2，则集束搜索的过程如下图："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://img-blog.csdn.net/20180414113522371?watermark/2/text/aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2d1b2xpbmRvbmdnbGQ=/font/5a6L5L2T/fontsize/400/fill/I0JBQkFCMA==/dissolve/70)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "得到第一个输出的概率分布`[0.1,0.1,0.3,0.4,0.1] [0.1,0.1,0.3,0.4,0.1]`，选择概率最大的前两个，`0.3`和`0.4`，即`Je`和`moi`。\n",
    "\n",
    "然后`Je`和`moi`分别作为`Decoder`的输入，得到两个概率分布，然后再选择概率和最大的前两个序列，`0.3+0.8`和`0.4+0.6`，即`Je suis`和`moi suis`。\n",
    "\n",
    "以此类推，最终可以得到两个序列，即`Je suis étudiant`和`moi suis étudiant`，很明显前者的概率和最大，为`2.2`，所以这个序列是最终得到的结果。\n",
    "\n",
    "集束搜索本质上也是贪心的思想，只不过它考虑了更多的候选搜索空间，因此可以得到更多的翻译结果。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://img-blog.csdn.net/20181011144011354?watermark/2/text/aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzE2MjM0NjEz/font/5a6L5L2T/fontsize/400/fill/I0JBQkFCMA==/dissolve/70)\n",
    "集束搜索可以认为是维特比算法的贪心形式，在维特比所有中由于利用动态规划导致当字典较大时效率低，而集束搜索使用beam size参数来限制在每一步保留下来的可能性词的数量。集束搜索是在测试阶段为了获得更好准确性而采取的一种策略，在训练阶段无需使用。\n",
    "\n",
    "假设字典为[a,b,c]，beam size选择2，则如下图有：\n",
    "\n",
    "1：在生成第1个词的时候，选择概率最大的2个词，那么当前序列就是a或b\n",
    "\n",
    "2：生成第2个词的时候，我们将当前序列a或b，分别与字典中的所有词进行组合，得到新的6个序列aa ab ac ba bb bc,然后从其中选择2个概率最高的，作为当前序列，即ab或bb\n",
    "\n",
    "3：不断重复这个过程，直到遇到结束符为止。最终输出2个概率最高的序列。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 加载训练好的模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The autoreload extension is already loaded. To reload it, use:\n",
      "  %reload_ext autoreload\n"
     ]
    }
   ],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "import os\n",
    "import sys\n",
    "sys.path.append('../')  # 返回notebook的上一级目录\n",
    "# sys.path.append('E:\\GitHub\\QA-abstract-and-reasoning')  # 效果同上"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'2.0.0'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.ticker as ticker\n",
    "np.set_printoptions(suppress=True)\n",
    "from utils.plot import plot_attention\n",
    "from utils.saveLoader import *\n",
    "from utils.config import *\n",
    "from layers import *\n",
    "from preprocess import Preprocess\n",
    "from gensim.models.word2vec import LineSentence, Word2Vec\n",
    "import tensorflow as tf\n",
    "# from model_layer import seq2seq_model\n",
    "import time\n",
    "tf.__version__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 Physical GPUs, 1 Logical GPUs\n"
     ]
    }
   ],
   "source": [
    "from utils.config_gpu import config_gpu\n",
    "config_gpu()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_x,train_y,test_x = load_train_dataset()  # 数据集\n",
    "vocab,vocab_reversed = load_vocab(VOCAB_PAD)  # vocab\n",
    "embedding_matrix = np.loadtxt(EMBEDDING_MATRIX_PAD)  # 预训练层"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'vocab_size': 32247, 'max_enc_len': 460, 'max_dec_len': 52, 'embed_size': 300, 'enc_units': 256, 'attn_units': 10, 'dec_units': 256, 'batch_size': 32, 'epochs': 6}\n"
     ]
    }
   ],
   "source": [
    "params = {}\n",
    "params[\"vocab_size\"] = len(vocab)\n",
    "params[\"max_enc_len\"] = train_x.shape[1]  # 260\n",
    "params[\"max_dec_len\"] = train_y.shape[1]  # 33\n",
    "params[\"embed_size\"] = embedding_matrix.shape[1]\n",
    "params[\"enc_units\"] = 256\n",
    "params[\"attn_units\"] = 10\n",
    "params[\"dec_units\"] = params[\"enc_units\"]\n",
    "params[\"batch_size\"] = 32\n",
    "params[\"epochs\"] = 4\n",
    "print(params)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from seq2seq import *\n",
    "model=Seq2Seq(params)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 载入模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'vocab_size': 32247,\n",
       " 'max_enc_len': 460,\n",
       " 'max_dec_len': 52,\n",
       " 'embed_size': 300,\n",
       " 'enc_units': 256,\n",
       " 'attn_units': 10,\n",
       " 'dec_units': 256,\n",
       " 'batch_size': 32,\n",
       " 'epochs': 6}"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<seq2seq.Seq2Seq at 0x209ae5e5c08>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2 读取训练好的模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model restored\n"
     ]
    }
   ],
   "source": [
    "from utils.config import CKPT_DIR, CKPT_PREFIX\n",
    "from utils.saveLoader import del_all_files_of_dir\n",
    "# 为了开始重新训练而不是继续上次的训练\n",
    "# del_all_files_of_dir(CKPT_DIR)\n",
    "ckpt = tf.train.Checkpoint(Seq2Seq=model)\n",
    "ckpt_manager = tf.train.CheckpointManager(ckpt, CKPT_DIR, max_to_keep=5)\n",
    "ckpt.restore(ckpt_manager.latest_checkpoint)\n",
    "print(\"Model restored\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# code "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Hypothesis:\n",
    "    \"\"\" Class designed to hold hypothesises throughout the beamSearch decoding \"\"\"\n",
    "    def __init__(self, tokens, log_probs, hidden, attn_dists):\n",
    "        self.tokens = tokens  # list of all the tokens from time 0 to the current time step t\n",
    "        self.log_probs = log_probs  # list of the log probabilities of the tokens of the tokens\n",
    "        self.hidden = hidden  # decoder hidden state after the last token decoding\n",
    "        self.attn_dists = attn_dists  # attention dists of all the tokens\n",
    "        self.abstract = \"\"\n",
    "\n",
    "    def extend(self, token, log_prob, hidden, attn_dist):\n",
    "        \"\"\"Method to extend the current hypothesis by adding the next decoded token and all the informations associated with it\"\"\"\n",
    "        return Hypothesis(tokens=self.tokens + [token],  # we add the decoded token\n",
    "                          log_probs=self.log_probs + [log_prob],  # we add the log prob of the decoded token\n",
    "                          hidden=hidden,  # we update the state\n",
    "                          attn_dists=self.attn_dists + [attn_dist])\n",
    "    @property\n",
    "    def latest_token(self):\n",
    "        return self.tokens[-1]\n",
    "\n",
    "    @property\n",
    "    def tot_log_prob(self):\n",
    "        return sum(self.log_probs)\n",
    "\n",
    "    @property\n",
    "    def avg_log_prob(self):\n",
    "        return self.tot_log_prob / len(self.tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 初始化一个对象列表"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(32, 460)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "enc_inp = train_x[:params[\"batch_size\"]]\n",
    "enc_inp.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(TensorShape([32, 460, 256]), TensorShape([32, 256]))"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "enc_output, enc_hidden = model.call_encoder(enc_inp)\n",
    "enc_output.shape, enc_hidden.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "hyps = [Hypothesis(tokens=[vocab['<START>']],\n",
    "                   log_probs=[0.0],\n",
    "                   hidden=enc_hidden[0],\n",
    "                   attn_dists=[],\n",
    "                   ) for _ in range(params['batch_size'])]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[32243]"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hyps[0].tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "32"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(hyps)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "results = []  # list to hold the top beam_size hypothesises\n",
    "steps = 0  # initial step"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 获取最新tokens "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "latest_tokens = [h.latest_token for h in hyps]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "32\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[32243, 32243, 32243, 32243, 32243]"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(len(latest_tokens))\n",
    "latest_tokens[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 隐藏层状态"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "32"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hiddens = [h.hidden for h in hyps]\n",
    "len(hiddens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://img-blog.csdn.net/20180414103300419)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 单步运行decode"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "pad_index = vocab['<PAD>']\n",
    "unk_index = vocab['<UNK>']\n",
    "start_index = vocab['<START>']\n",
    "stop_index = vocab['<STOP>']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 第一轮\n",
    "steps=0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "def decoder_onestep(enc_output,dec_input,dec_hidden):\n",
    "    # 单个时间步 运行\n",
    "    preds, dec_hidden, context_vector,attention_weights = model.call_decoder_onestep(dec_input,dec_hidden, enc_output)\n",
    "    # 拿到top k个index 和 概率\n",
    "    top_k_probs, top_k_ids = tf.nn.top_k(tf.squeeze(preds), k=params[\"beam_size\"])\n",
    "    # 计算log概率\n",
    "    top_k_log_probs = tf.math.log(top_k_probs)\n",
    "    # 返回需要保存的中间结果和概率\n",
    "    return preds,dec_hidden,context_vector,attention_weights,top_k_log_probs,top_k_ids"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1, 32247)"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(1,params[\"vocab_size\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 单次搜索"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 计算第encoder的输出\n",
    "enc_output, enc_hidden = model.call_encoder(enc_inp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 第一个decoder输入 开始标签\n",
    "dec_input = tf.expand_dims(latest_tokens, 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 第一个隐藏层输入\n",
    "dec_hidden = enc_hidden"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 单步运行\n",
    "params[\"beam_size\"] = 3\n",
    "preds, dec_hidden, context_vector,attention_weights, top_k_log_probs, top_k_ids = decoder_onestep(enc_output,dec_input,dec_hidden)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 现阶段全部可能情况\n",
    "all_hyps = []\n",
    "# 原有的可能情况数量\n",
    "num_orig_hyps = 1 if steps == 0 else len(hyps)\n",
    "\n",
    "# 遍历添加所有可能结果\n",
    "for i in range(num_orig_hyps):\n",
    "    h, new_hidden, attn_dist = hyps[i], dec_hidden[i], attention_weights[i]\n",
    "    # 分裂 添加 beam size 种可能性\n",
    "    for j in range(params['beam_size']):\n",
    "        # 构造可能的情况\n",
    "        new_hyp = h.extend(token = top_k_ids[i, j].numpy(),\n",
    "                                       log_prob = top_k_log_probs[i, j],\n",
    "                                       hidden = new_hidden,\n",
    "                                       attn_dist = attn_dist)\n",
    "        # 添加可能情况\n",
    "        all_hyps.append(new_hyp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.0, <tf.Tensor: id=1020, shape=(), dtype=float32, numpy=1.9014189>]"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_hyps[1].log_probs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<__main__.Hypothesis at 0x7ff10c7eee80>,\n",
       " <__main__.Hypothesis at 0x7ff10c7ee5f8>,\n",
       " <__main__.Hypothesis at 0x7ff10c7eeac8>]"
      ]
     },
     "execution_count": 139,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_hyps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 排序"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 重置\n",
    "hyps = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 按照概率来排序\n",
    "sorted_hyps = sorted(all_hyps, key=lambda h: h.avg_log_prob, reverse=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.0, <tf.Tensor: id=1012, shape=(), dtype=float32, numpy=2.115679>]"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sorted_hyps[0].log_probs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 筛选top前beam_size句话 top 3\n",
    "for h in sorted_hyps:\n",
    "    if h.latest_token == stop_index:\n",
    "        # 长度符合预期,遇到句尾,添加到结果集\n",
    "        if steps >= params['min_dec_steps']:\n",
    "            results.append(h)\n",
    "    else:\n",
    "        # 未到结束 ,添加到假设集\n",
    "        hyps.append(h)\n",
    "    \n",
    "    # 如果假设句子正好等于beam_size 或者结果集正好等于beam_size 就不在添加\n",
    "    if len(hyps) == params['beam_size'] or len(results) == params['beam_size']:\n",
    "        break\n",
    "steps += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "if len(results) == 0:\n",
    "    results = hyps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.0, <tf.Tensor: id=1012, shape=(), dtype=float32, numpy=2.115679>]"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results[0].log_probs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: id=1052, shape=(), dtype=float32, numpy=1.0578395>"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results[0].avg_log_prob"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [],
   "source": [
    "hyps_sorted = sorted(results, key=lambda h: h.avg_log_prob, reverse=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_hyp = hyps_sorted[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_hyp.abstract = \" \".join([vocab_reversed[index] for index in best_hyp.tokens])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'<START> 你好'"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_hyp.abstract"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "References\n",
    "\n",
    "[1] https://www.tensorflow.org/tutorials/seq2seq\n",
    "\n",
    "[2] https://blog.csdn.net/guolindonggld/article/details/79938567"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# beam search 方法整合"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [],
   "source": [
    "def beam_decode(model,batch,vocab, params):\n",
    "    # 初始化mask\n",
    "    start_index = vocab['<START>']\n",
    "    stop_index = vocab['<STOP>']\n",
    "    \n",
    "    batch_size= params['batch_size']\n",
    "    \n",
    "    # 单步decoder\n",
    "    def decoder_onestep(enc_output,dec_input,dec_hidden):\n",
    "        # 单个时间步 运行\n",
    "        preds, dec_hidden, context_vector,attention_weights = model.call_decoder_onestep(dec_input,dec_hidden, enc_output)\n",
    "        # 拿到top k个index 和 概率\n",
    "        top_k_probs, top_k_ids = tf.nn.top_k(tf.squeeze(preds), k=params[\"beam_size\"])\n",
    "        # 计算log概率\n",
    "        top_k_log_probs = - tf.math.log(top_k_probs)\n",
    "        # 返回需要保存的中间结果和概率\n",
    "        return preds,dec_hidden,context_vector,attention_weights,top_k_log_probs,top_k_ids\n",
    "    \n",
    "    # 计算第encoder的输出\n",
    "    enc_output, enc_hidden = model.call_encoder(batch)\n",
    "    \n",
    "    # 初始化batch size个 假设对象\n",
    "    hyps = [Hypothesis(tokens=[start_index],\n",
    "                   log_probs=[0.0],\n",
    "                   hidden=enc_hidden[0],\n",
    "                   attn_dists=[],\n",
    "                   ) for _ in range(batch_size)]\n",
    "    # 初始化结果集\n",
    "    results = []  # list to hold the top beam_size hypothesises\n",
    "    # 遍历步数\n",
    "    steps = 0  # initial step\n",
    "    \n",
    "    # 第一个隐藏层输入\n",
    "    dec_hidden = enc_hidden\n",
    "    \n",
    "    \n",
    "    # 长度还不够 并且 结果还不够 继续搜索\n",
    "    while steps < params['max_dec_steps'] and len(results) < params['beam_size']:\n",
    "        # 获取最新待使用的token\n",
    "        latest_tokens = [h.latest_token for h in hyps]\n",
    "        # 获取所以隐藏层状态\n",
    "        hiddens = [h.hidden for h in hyps]\n",
    "        # 最新输入\n",
    "        dec_input = tf.expand_dims(latest_tokens, 1)\n",
    "        \n",
    "        # 单步运行decoder 计算需要的值\n",
    "        preds, dec_hidden, context_vector,attention_weights, top_k_log_probs, top_k_ids = decoder_onestep(enc_output,dec_input,dec_hidden)\n",
    "        \n",
    "        # 现阶段全部可能情况\n",
    "        all_hyps = []\n",
    "        # 原有的可能情况数量\n",
    "        num_orig_hyps = 1 if steps == 0 else len(hyps)\n",
    "\n",
    "        # 遍历添加所有可能结果\n",
    "        for i in range(num_orig_hyps):\n",
    "            h, new_hidden, attn_dist = hyps[i], dec_hidden[i], attention_weights[i]\n",
    "            # 分裂 添加 beam size 种可能性\n",
    "            for j in range(params['beam_size']):\n",
    "                if params['batch_size']==1:\n",
    "                    # 构造可能的情况\n",
    "                    new_hyp = h.extend(token = top_k_ids[j].numpy(),\n",
    "                                       log_prob = top_k_log_probs[j],\n",
    "                                       hidden = new_hidden,\n",
    "                                       attn_dist = attn_dist)\n",
    "                else:\n",
    "                    # 构造可能的情况\n",
    "                    new_hyp = h.extend(token = top_k_ids[i, j].numpy(),\n",
    "                                       log_prob = top_k_log_probs[i, j],\n",
    "                                       hidden = new_hidden,\n",
    "                                       attn_dist = attn_dist)\n",
    "                # 添加可能情况\n",
    "                all_hyps.append(new_hyp)\n",
    "        \n",
    "        # 重置\n",
    "        hyps = []\n",
    "        # 按照概率来排序\n",
    "        sorted_hyps = sorted(all_hyps, key=lambda h: h.avg_log_prob, reverse=True)\n",
    "        \n",
    "        # 筛选top前beam_size句话\n",
    "        for h in sorted_hyps:\n",
    "            if h.latest_token == stop_index:\n",
    "                # 长度符合预期,遇到句尾,添加到结果集\n",
    "                if steps >= params['min_dec_steps']:\n",
    "                    results.append(h)\n",
    "            else:\n",
    "                # 未到结束 ,添加到假设集\n",
    "                hyps.append(h)\n",
    "\n",
    "            # 如果假设句子正好等于beam_size 或者结果集正好等于beam_size 就不在添加\n",
    "            if len(hyps) == params['beam_size'] or len(results) == params['beam_size']:\n",
    "                break\n",
    "\n",
    "        steps += 1\n",
    "        \n",
    "    if len(results) == 0:\n",
    "        results = hyps\n",
    "    \n",
    "    hyps_sorted = sorted(results, key=lambda h: h.avg_log_prob, reverse=True)\n",
    "    best_hyp = hyps_sorted[0]\n",
    "    best_hyp.abstract = \" \".join([reverse_vocab[index] for index in best_hyp.tokens])\n",
    "    return best_hyp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 读取模型\n",
    "model = Seq2Seq(params)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 构造数据\n",
    "test_batch=test_x[:params[\"batch_size\"]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(32, 460)"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_batch.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "ename": "InvalidArgumentError",
     "evalue": "ConcatOp : Dimensions of inputs should match: shape[0] = [32,1,256] vs. shape[1] = [3,1,300] [Op:ConcatV2] name: concat",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mInvalidArgumentError\u001b[0m                      Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-92-8b864390027e>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      2\u001b[0m \u001b[0mparams\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m'min_dec_steps'\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;36m4\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      3\u001b[0m \u001b[0mparams\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m'max_dec_steps'\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;36m50\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 4\u001b[1;33m \u001b[0mbest_hyp\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mbeam_decode\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mmodel\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0mtest_batch\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0mvocab\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mparams\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;32m<ipython-input-87-d58da54a1397>\u001b[0m in \u001b[0;36mbeam_decode\u001b[1;34m(model, batch, vocab, params)\u001b[0m\n\u001b[0;32m     45\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     46\u001b[0m         \u001b[1;31m# 单步运行decoder 计算需要的值\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 47\u001b[1;33m         \u001b[0mpreds\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mdec_hidden\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mcontext_vector\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0mattention_weights\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mtop_k_log_probs\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mtop_k_ids\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mdecoder_onestep\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0menc_output\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0mdec_input\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0mdec_hidden\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     48\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     49\u001b[0m         \u001b[1;31m# 现阶段全部可能情况\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m<ipython-input-87-d58da54a1397>\u001b[0m in \u001b[0;36mdecoder_onestep\u001b[1;34m(enc_output, dec_input, dec_hidden)\u001b[0m\n\u001b[0;32m      9\u001b[0m     \u001b[1;32mdef\u001b[0m \u001b[0mdecoder_onestep\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0menc_output\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0mdec_input\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0mdec_hidden\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     10\u001b[0m         \u001b[1;31m# 单个时间步 运行\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 11\u001b[1;33m         \u001b[0mpreds\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mdec_hidden\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mcontext_vector\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0mattention_weights\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mmodel\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mcall_decoder_onestep\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdec_input\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0mdec_hidden\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0menc_output\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     12\u001b[0m         \u001b[1;31m# 拿到top k个index 和 概率\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     13\u001b[0m         \u001b[0mtop_k_probs\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mtop_k_ids\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mtf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mnn\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mtop_k\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mtf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msqueeze\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mpreds\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mk\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mparams\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m\"beam_size\"\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mE:\\GitHub\\QA-abstract-and-reasoning\\seq2seq.py\u001b[0m in \u001b[0;36mcall_decoder_onestep\u001b[1;34m(self, dec_input, dec_hidden, enc_output)\u001b[0m\n\u001b[0;32m     34\u001b[0m                                         \u001b[1;32mNone\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     35\u001b[0m                                         \u001b[1;32mNone\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 36\u001b[1;33m                                         context_vector)\n\u001b[0m\u001b[0;32m     37\u001b[0m         \u001b[1;32mreturn\u001b[0m \u001b[0mpred\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mdec_hidden\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mcontext_vector\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mattention_weights\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     38\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mE:\\software\\Anaconda3\\envs\\tf2.0\\lib\\site-packages\\tensorflow_core\\python\\keras\\engine\\base_layer.py\u001b[0m in \u001b[0;36m__call__\u001b[1;34m(self, inputs, *args, **kwargs)\u001b[0m\n\u001b[0;32m    889\u001b[0m           with base_layer_utils.autocast_context_manager(\n\u001b[0;32m    890\u001b[0m               self._compute_dtype):\n\u001b[1;32m--> 891\u001b[1;33m             \u001b[0moutputs\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mcall\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mcast_inputs\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m*\u001b[0m\u001b[0margs\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    892\u001b[0m           \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_handle_activity_regularization\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0minputs\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0moutputs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    893\u001b[0m           \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_set_mask_metadata\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0minputs\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0moutputs\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0minput_masks\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mE:\\GitHub\\QA-abstract-and-reasoning\\layers.py\u001b[0m in \u001b[0;36mcall\u001b[1;34m(self, x, hidden, enc_output, context_vector)\u001b[0m\n\u001b[0;32m     74\u001b[0m         \u001b[1;31m# attention_weights = self.attention(hidden, enc_output)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     75\u001b[0m         \u001b[0mx\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0membedding\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mx\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 76\u001b[1;33m         \u001b[0mx\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mtf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mconcat\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0mtf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mexpand_dims\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mcontext_vector\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;36m1\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mx\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0maxis\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;33m-\u001b[0m\u001b[1;36m1\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     77\u001b[0m         \u001b[0moutput\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mstate\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mgru\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mx\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     78\u001b[0m         \u001b[0moutput\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mtf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mreshape\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0moutput\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m(\u001b[0m\u001b[1;33m-\u001b[0m\u001b[1;36m1\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0moutput\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;36m2\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mE:\\software\\Anaconda3\\envs\\tf2.0\\lib\\site-packages\\tensorflow_core\\python\\util\\dispatch.py\u001b[0m in \u001b[0;36mwrapper\u001b[1;34m(*args, **kwargs)\u001b[0m\n\u001b[0;32m    178\u001b[0m     \u001b[1;34m\"\"\"Call target, and fall back on dispatchers if there is a TypeError.\"\"\"\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    179\u001b[0m     \u001b[1;32mtry\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 180\u001b[1;33m       \u001b[1;32mreturn\u001b[0m \u001b[0mtarget\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m*\u001b[0m\u001b[0margs\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    181\u001b[0m     \u001b[1;32mexcept\u001b[0m \u001b[1;33m(\u001b[0m\u001b[0mTypeError\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mValueError\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    182\u001b[0m       \u001b[1;31m# Note: convert_to_eager_tensor currently raises a ValueError, not a\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mE:\\software\\Anaconda3\\envs\\tf2.0\\lib\\site-packages\\tensorflow_core\\python\\ops\\array_ops.py\u001b[0m in \u001b[0;36mconcat\u001b[1;34m(values, axis, name)\u001b[0m\n\u001b[0;32m   1429\u001b[0m           dtype=dtypes.int32).get_shape().assert_has_rank(0)\n\u001b[0;32m   1430\u001b[0m       \u001b[1;32mreturn\u001b[0m \u001b[0midentity\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mvalues\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;36m0\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mname\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mname\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 1431\u001b[1;33m   \u001b[1;32mreturn\u001b[0m \u001b[0mgen_array_ops\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mconcat_v2\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mvalues\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mvalues\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0maxis\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0maxis\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mname\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mname\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   1432\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1433\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mE:\\software\\Anaconda3\\envs\\tf2.0\\lib\\site-packages\\tensorflow_core\\python\\ops\\gen_array_ops.py\u001b[0m in \u001b[0;36mconcat_v2\u001b[1;34m(values, axis, name)\u001b[0m\n\u001b[0;32m   1247\u001b[0m       \u001b[1;32melse\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1248\u001b[0m         \u001b[0mmessage\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0me\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mmessage\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 1249\u001b[1;33m       \u001b[0m_six\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mraise_from\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0m_core\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_status_to_exception\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0me\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mcode\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mmessage\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;32mNone\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   1250\u001b[0m   \u001b[1;31m# Add nodes to the TensorFlow graph.\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1251\u001b[0m   \u001b[1;32mif\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[0misinstance\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mvalues\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m(\u001b[0m\u001b[0mlist\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mtuple\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mE:\\software\\Anaconda3\\envs\\tf2.0\\lib\\site-packages\\six.py\u001b[0m in \u001b[0;36mraise_from\u001b[1;34m(value, from_value)\u001b[0m\n",
      "\u001b[1;31mInvalidArgumentError\u001b[0m: ConcatOp : Dimensions of inputs should match: shape[0] = [32,1,256] vs. shape[1] = [3,1,300] [Op:ConcatV2] name: concat"
     ]
    }
   ],
   "source": [
    "# 获得最好的语句\n",
    "params['min_dec_steps']=4\n",
    "params['max_dec_steps']=50\n",
    "best_hyp=beam_decode(model,test_batch,vocab, params)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'<START> 左高右 难关 齿合器 某一 第一段 齿合器 某一 第一段 齿合器 某一 第一段 齿合器 某一 第一段 齿合器 某一 第一段 齿合器 某一 第一段 齿合器 某一 第一段 齿合器 某一 第一段 齿合器 某一 第一段 齿合器 某一 第一段 齿合器 某一 第一段 齿合器 某一 第一段 齿合器 某一 第一段 齿合器 某一 第一段 齿合器 某一 第一段 齿合器 某一 第一段'"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_hyp.abstract"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:tf2.0]",
   "language": "python",
   "name": "conda-env-tf2.0-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
